Skip to content

Update gateway deployment/onboarding documentation - #157

Merged
floormind merged 1 commit into
mainfrom
docs/promote-preprod
Jul 14, 2026
Merged

Update gateway deployment/onboarding documentation#157
floormind merged 1 commit into
mainfrom
docs/promote-preprod

Conversation

@carlosmartinez

@carlosmartinez carlosmartinez commented Jun 30, 2026

Copy link
Copy Markdown
Contributor

Captures the operational knowledge from the first production gateway onboarding, promotion and live clinics:

  • onboard-hospital-vm.md — new "Who can run this" permissions section, Step 0 (temporary onboarding secret), two new troubleshooting entries for the subscription misconfigurations that blocked prod onboarding, a pointer to where the relay namespace comes from, and Step 6 rewritten as acceptance criteria (services + ports + relay log shows Connected - waiting – a Running service that never connects is the failure mode that matters; the old Log Analytics heartbeat check is unreliable while VM telemetry export is broken).
  • promote-vm-preprod-to-prod.md — cleanup verification, the Gateway.oid update step, end-to-end check via the Gateway actions admin, and four new gotchas from the incident.
  • live-clinic-debugging.md (new) — symptom → check mapping for every hop of the pipeline, VM-side and Azure-side, plus the echo probe section.
  • scripts/diagnostics/relay_echo_probe.py (new) — tests relay → gateway independently of Manage, from any laptop or from inside a Manage container.
  • scripts/powershell/debug_toolkit.ps1 (new) — VM-side query/tail/health helpers for clinic-day support.

Content merged from the DevOps team's onboarding write-ups (acceptance criteria adopted nearly verbatim) plus the debug session notes. Identifiers are placeholders throughout.

Review notes — two accuracy questions for @floormind

  1. The original notes said "the bicep is running in the hub subscription" — I've neutralised this to "deployment tool against the hub subscription and writes to the spoke". If there is an actual Bicep deployment in the picture, let's name it
  2. The Monitoring Contributor requirement for the pipeline MI is kept but reframed as "assigned by infrastructure code, no hand" — please confirm that matches where it's actually assigned.

Onboarding: add the permissions the engineer actually needs (PIM for
hub and spoke, enterprise app ownership), the temporary-secret
procedure and its 24-hour trap, the two subscription misconfigurations
that blocked the first prod onboarding (missing Azure Connected Machine
Onboarding role, unregistered resource providers), where the relay
namespace is provisioned, and a proper acceptance-criteria verification
step - including that a Running service which never logs "Connected -
waiting" is not healthy.

Promotion: the cleanup verify now has commands and a warning (pre-prod
data survived the first promotion onto the prod gateway and was found a
month later during a live clinic); re-onboarding creates a new managed
identity so Gateway.oid must be updated or every image upload 403s; the
end-to-end check goes through the Gateway actions admin; the gotchas
table gains the failure modes we actually hit.

Also, add live-clinic debugging runbook and diagnostic tooling.

Co-authored-by: Ife Ayelabola <floormind@users.noreply.github.com>
@carlosmartinez carlosmartinez changed the title Runbook for promoting a Gateway VM from pre-prod to prod Update gateway deployment/onboarding documentation Jul 10, 2026
@floormind

Copy link
Copy Markdown
Contributor

@floormind
floormind merged commit 28f6918 into main Jul 14, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants